DOCUMENT RESUME 



ED 445 628 



HE 033 301 



AUTHOR 

TITLE 

PUB DATE 
NOTE 



PUB TYPE 
EDRS PRICE 
DESCRIPTORS 



IDENTIFIERS 



Yogan, Lissa J. ; Mercer-Taylor , Beth 

Developing Faculty as Stakeholders in Assessment: A Case 
Study. 

2000 - 00-00 

2 Ip . ; Paper presented at the Annual Forum of the Association 
for Institutional Research (40th, Cincinnati, OH, May 21-24, 
2000 ) . 

Reports - Research (143) -- Speeches/Meeting Papers (150) 

MF01/PC01 Plus Postage. 

College Faculty; College Freshmen; Educational Environment; 
Educational Research; *Evaluation Methods; * Formative 
Evaluation; Higher Education; Participant Satisfaction; 
*Program Evaluation; School Culture; Student Attitudes; 
Teacher Participation 
*AIR Forum 



ABSTRACT 



This case study examined formative assessment of a 
first-year core course initiated in response to an external requirement. The 
paper discusses the campus assessment climate; describes how course faculty 
became proponents for assessment; explains how the core survey became housed 
in a larger student satisfaction survey; and outlines key elements of 
successful assessment programs. A researcher teamed with faculty to conduct 
the evaluation. The study examined the traditional knowledge and role of 
faculty in institutional assessment. Faculty, who were initially ambivalent 
or against formal assessment, were the primary audience. Qualitative methods 
were used to overcome faculty resistance. A research instrument was developed 
to assess the core program and students’ satisfaction with their first-year 
experience. Nearly 600 freshmen completed the survey; student focus groups 
were also conducted. Assessment results were delivered during a faculty 
workshop. Faculty were given time to discuss questions and results. All 
instructors received printouts of their own and group results. Key elements 
of this project's success in developing faculty as stakeholders in assessment 
included: creating an interdisciplinary assessment team of faculty and staff; 
developing trust by listening to and respecting student and faculty 
viewpoints; and letting faculty adapt assessment to their own needs. 
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Developing Faculty as Stakeholders in Assessment: A Case Study 



Abstract 

A researcher at a private university teamed with two faculty members to assess a 
required, general education course. The assessment was formative rather than 
summative, or designed to inform activity in progress rather than judge ultimate 
success or failure. Creating a successful process demanded that the researchers 
not only utilize sound research practices but also examine the traditional 
knowledge and role of faculty in institutional assessment. Faculty, who were 
initially ambivalent or against 'formal assessment,' were the primary audience. 
The process through which faculty become assessment stakeholders will be 
outlined, as will general assessment “dos and don’ts” for working with faculty or 
other apprehensive audiences. 
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Developing Faculty as Stakeholders in Assessment: A Case Study 



Introduction 

Assessment of a first year general education course at a small, private 
university was initiated in response to an external requirement. In 1996, an 
external grant provided funding to develop a year-long, required, interdisciplinary, 
10 credit, "Core” course. The grant-maker required serious attention to 
assessment. In addition to the familiar summative assessment, which evaluates 
successes and failures of a fully implemented program, the grant-maker asked 
the university to engage in formative assessment. Formative assessment was to 
be incorporated into the Core’s development and implementation, formalizing 
reflection about goals and objectives and offering opportunities to modify the 
course mid-stream. 

This narrative briefly discusses the assessment climate on campus, 
describes how Core faculty became proponents for assessment, how the Core 
survey came to be housed in a larger student satisfaction-survey and outlines 
key elements of a successful assessment project. While the focus is on lessons 
learned at one institution, the presentation will incorporate theoretical literature, 
knowledge of faculty academic training and an analysis of colleges and 
universities’ structures in discussing key elements that can be applied to 
institutions of any size. 
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Assessment Climate on Campus 

Since instructional faculty at this campus often dismissed assessment as 
a dubious bureaucratic exercise, instituting meaningful formative assessment 
— meanta change in culture. Assessment itself was not new, but formative 
assessment designed for faculty use was. Although an assessment plan with 
some mechanism for improving programs was and still is required for each 
department, faculty regarded administrators as the key audience. Other 
assessment programs included a five-year program review, accreditation efforts, 
the Comprehensive Institutional Research Project (CIRP) survey, and Career 
Center surveys of recent alumni. These programs also primarily served 
administrators or external audiences. The faculty-directed teaching and learning 
center on campus has sponsored luncheons on classroom assessment 
techniques, but attendance has been poor. Faculty members typically cited 
their overwhelming teaching loads (four classes per semester) and/or a belief 
that 'numbers' could not really measure the processes of teaching and learning 
as reasons not to become involved in classroom assessment. 

Their reasons for not doing formal, quantitative assessments are not 
unusual. An American College Testing (ACT) survey of two and four-year public 
colleges found that over half of those who responded said that involving faculty 
in assessment was a significant problem (Steele 1996). The reasons faculty give 
for not wanting to do assessment differ a little from campus to campus yet 
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Schilling and Schilling (1998, p. 17 - 22) report that the concerns across all 
campuses can be reduced to a list of twelve. The twelve concerns that they cite 
are: 

1 . We already do it . (refers to faculty's role in assessing student 
achievement, i.e. grades). 

2. The data will be misused, (refers to campus politics and the potential 
for data to be used to against them or not be used at all) 

3. I'm afraid of change, (what will be asked of me next?) 

4. The criteria are unclear, (how will you judge my work? Will I be one of 
the 'bad ones'?) 

5. Assessment violates my academic freedom. (Will this work begin to 
shape or alter how I plan and teach my courses?) 

6. Assessment is inconsistent with academic values. ( Complexity of 
work not believed to be captured and no appreciation of academic 
expertise.) 

7. Faculty lack knowledge of assessment, (really do not know what this 
work entails and/or how to do it.) 

8. I have no confidence in existing instruments. (They can identify holes 
in current instruments or pinpoint areas that are not measured.) 

9. Too often, what is tested becomes what is valued. . (It is hard to 
measure character growth, internal cognitive process involved in 
analyzing literature etc.) 




0 



6 



5 



10. We don't need more bureacracy. (Believe resources used for 



assessment could be better used in teaching/learning process.) 

11. My plate is too full. (Many already feel overburdened by teaching, 
research, committee and advising demands.) 

12. Nobody told me about the shift from teachers and teaching to students 
and learning. (May disagree with with premise that we need to focus 
on what students have learned; may not have been included in 
discussions that change focus from teaching to student learning.) 

There were over 30 faculty members involved in teaching the Core 
program. Most of the faculty came from the College of Arts and Sciences but 
there were also a few from the College of Business and the College of 
Engineering. Either in informal or formal discussions the researchers reported 
hearing at least 10 of the 12 reasons cited by Schilling and Schilling. Setting up 
an assessment procedure that faculty would use to improve courses seemed a 
daunting task. 

Key Step #1: Creating an Assessment Team 

A team of several Core faculty, the university’s institutional researcher, 
and two faculty consultants took up the challenge of developing a formative Core 
assessment program. Ferren reports, "Assessment programs that do not 
emerge from the unique needs of the institution and the specific capabilities of 
the faculty who are to carry them out and respond to the findings run into 
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significant resistance ... To get commitment, faculty must be involved at the 
outset (1993, p. 5). 

The Core faculty involved in this effort included the Core Director and 
members of the Core Steering Committee. Senior, tenured faculty and more 
recently hired faculty sat together on the Core Steering Committee and made 
final decisions on texts and syllabus for this important new University program. 
The research team was composed of an independent higher education evaluator 
who taught part-time in the Core program, a Sociology professor, who formerly 
worked as a survey research consultant, and the University's institutional 
researcher. 

The survey questions related to the Core were housed in a larger 
instrument that attempted to capture the essence of students' first year 
experience. However, since the Core program had provided the impetus for the 
assessment project, the research team agreed that Core faculty would be the 
primary audience for assessment results and first to receive reports. Secondary 
audiences would be the University retention committee, the Provost’s office, the 
assessment committee r the teaching-and learning-center, the chapel council, the 
honor’s college, and the Athletics department. 

The Core committee and research team began to work together. One 
member described the work as "exciting". She went on to comment that such a 
project was the first time she had experienced a true joining together of diverse 
institutional constituents. Her feelings and the work mirrored the words of 
Tarule and Tereault, "In its best form, assessment contributes to the life of the 
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academy in a way that promotes further thought, interpersonal and professional 
connections, and enhanced student development opportunities. To begin any 
assessment project is to enter into a conversation about all the important issues 
in education" (1992, pp. 49-50). 

Key Step #2: Using Qualitative Methods to Overcome Faculty Resistance 

The assessment team’s first step was discovering student and faculty 
impressions of the former first-year required courses, as well as hopes and fears 
for the new Core course and experience. Deliberately, this process was not 
rushed. Over the course of eight months, qualitative methods were employed to 
gather the needed information. 

Social researchers systematically collect and analyze empirical 
evidence in order to understand and explain social life. But a 
qualitative researcher goes about this differently than a quantitative 
researcher does. The most obvious difference is that qualitative 
data tend to be in the form of words, sentences, and paragraphs 
rather than numbers (Neuman, 1991 p. 321) 

The research team felt strongly that it was best to begin by listening. They 
listened to students and then they talked and listened to all Core faculty. They 
did this during convenient times. No faculty member had to give up designated 
work time to talk with the researchers. Students were invited to talk at times that 
did not conflict with class schedules. 
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In the spring of 1998, twenty-two freshmen were interviewed for forty-five 
minutes each by two female members of the research team. 1 The students 
communicated a preference for discussion-based teaching to lectures; they 
valued faculty whose concern for them went beyond the classroom, and had 
strong views about several texts. 

These and other findings-were.pnesented at the Core faculty workshop in 
May. As the faculty struggled to establish common texts and a syllabus, many 
felt overwhelmed by information. Yet, when asked by a member of the research 
team to write about and discuss “signs of success" and “signs of failure" for the 
Core, they referred to and even quoted the student interviews. 

Some of the student comments took on special meanings and were 
repeated by the faculty or repeatedly used to define specific concepts. Indeed 
the presentation of the qualitative student data gave rise to faculty discussion on 
assessment that was enthusiastic and enjoyable. Neuman writes that this is the 
typical effect of qualitative reports. "Most people find reports of qualitative 
research more enjoyable to read. Instead of a formal, neutral tone with statistics, 
qualitative reports often contain rich description, colorful detail, and unusual 
characters: they give the reader a feel for social settings (1991 , p. 322). 

In this way, the voices of articulate students helped shape Core goals and 
objectives and ease the faculty into the assessment process. As the faculty 
members voiced in small groups and in a larger group discussion, their hopes 
and fears about this new program, they unwittingly began to shape the final 



1 Ward and Grant (1985) and Grant, Ward, and Rong (1987) analyze research in sociology journals and 
suggest that women are more likely to use qualitative research techniques than men. 
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questionnaire and join the research team. The mood was upbeat and faculty 
were engaged in an optimistic exercise that gave voice to a cultural 
transformation. 

Reflexive assessment moves away from a culture of fear and 
retribution and toward an understanding of how to create a climate 
for improvement. In effect, I am suggesting that rather than 
develop an evaluation system that looks backward and determines 
how well someone performed, we create a structure that looks 
forward and tries to outline how the individual and organization 
want to peform (Tierney 1998, p. 55). 

The faculty conversation on hope and fears for the new program was very 
productive. Not only did individual members get a chance to be heard but the 
group as a whole articulated the culture they wished to create. A research team 
member recorded all that was said and produced a document that organized and 
summarized the faculty's comments. 

Over the summer, word about the Core workshop made its way around 
campus and piqued more faculty interest. Core faculty posed numerous 
questions, and by fall of 1999, agreed that a written survey instrument that would 
provide more representative information than interviews was needed. The 
culture surrounding assessment was beginning to change. The faculty had 
moved from negative attitudes toward research to a position of desiring more 
broad-based assessment tools. 
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This change in culture and faculty attitude goes against what many recent 
research studies have called faculty disengagement (Kennedy, 1997; Sacks, 
1996; Shapiro, 1997). Kluge (1993) wrote: 

Faculty have been encouraged to remove themselves, to 
disengage somewhat from a full, intense commitment to 
students. . . We're teaching less, there's less emphasis on faculty 
keeping office hours and attending meetings, more celebration of 
publication and other accomplishments. Students are aware of 
that, aware of it when you go from asking for four papers to asking 
for one, aware of it when you go from ten office hours a week to 
two. That suggests disengagement (p. 38). 

The faculty at this institution were committing not just to a writing intensive 
course that was discussion based and demanded that they meet with students 
individually over the course of the semester but to a new role in assessment. 
Faculty were engaged and becoming intricately involved in how the future of this 
program would be formed and evaluated. 

Key Step #3: Creating the Research Instrument 

Research on the advantages and disadvantages of commercial survey 
instruments and information from assessment conferences and literature led to 
the decision to develop a survey in-house. The goals of the survey were two 
fold: the first was to assess the Core program and the second was to assess 
student satisfaction with their first year experience. 



The instrument has 150 total questions broken into 1 1 categories that 
range from satisfaction with academic and technical support to satisfaction with 
campus programming. The 11 sections represent the diverse institutional arenas 
in which students interact and which affect their overall first year experience. 
"Assessment efforts need to cut across courses to make connections within the 
academic experience to better understand the degree to which attending college 
is meeting students' educational and social needs" (Sanders and Burton, 1996, 
p. 556). 

Because the survey was designed to serve two primary groups: Core faculty and 
members of the Retention Committee, care was taken to make sure both groups 
saw and were able to comment on consecutive drafts of the survey. Like the 
initital qualitative processes, this process was also time consuming. The 
creation of the instrument took four months and mandated numerous revisions. 

Like other institutions, this one had a loosely coupled retention network. 
The central belief of loosely coupled retention networks is usually that smaller 
units within the institution know their clientele the best (Kluepfel, Parelius, 
Roberts, 1994). Thus, the member of the research team charged with survey 
design met with a diverse group of people. They included members of residence 
life, representatives of the campus chapel, a student-athlete advisor, an 
engineering professor, the Director of Multi-cultural affairs, the Dean of Students, 
an employee who worked in financial aid, the head of the academic help 
program and students in a spring semester general education class. The 
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involvement of so many groups undoubtedly contributed to the survey's long 
length but also established a link between faculty and staff and students. 

Each group represented by a staff or faculty member were at first 
somewhat territorial, concerned almost solely with 'their section'. Thus an early 
draft of the questionnaire missed one of the most talked about components of 
first year college life: food! This oversight was quickly pointed out by the 
students in the general education class, underscoring the need for any 
assessment of college life to include students in all phases of the project. 

Key Step #4: Delivering the Results and Re-involving Faculty 

The university's first survey of all freshmen, administered in class during 
the second week of April, 1999, focused on the Core experience in the context of 
broad academic and student life issues. Nearly 600 students (90% response 
rate) completed a questionnaire that asked 150 questions about their first year 
experience. In addition to the questionnaire, other new methods of assessment 
were developed and some traditional forms retained as well. Focus groups 
involving thirty-three freshmen-were_held in late spring, using questions similar to 
those of the previous year’s interviews. The assessment team also provided 
faculty with grade distributions by section, gender, and race, and worked on a 
course evaluation with questions about specific texts and class exercises. 

Because some faculty were not receptive to quantitative measurements of 
teaching and learning and others were ambivalent about 'numbers-oriented' 
assessment, much care was taken to deliver the results of the survey and other 



assessment efforts in a way that faculty could “own" and use. The initial results 
were given during the end-of-the-year faculty workshop. This is a two-day 
workshop at which Core faculty gather to talk about the experience of the Core 
program and plan for the following year's changes. 

The presentation was interactive and given without a lot of statistical 
language. A comic poking fun at the validity of survey answers was the opening 
overhead. The report of initial results focused on key areas relating to the 
qualitative assessment of faculty's stated hopes and concerns for the course. 
Care was taken to not provide too much information too quickly. No statistical 
models were presented and indeed several faculty needed to be reminded of 
what some consider the most basic statistical terms i.e. mean, correlation, 
standard deviation. Differences between men and women and between those of 
color and those classified as white were highlighted. The role of GPA in 
students' willingness to express satisfaction was examined. 

Lively discussions took place over questions and results. The faculty 
debated: should they care more about students seeing the Core as their favorite 
-class? They attempted to make sense of the findings that that men were more 
comfortable participating in class discussions than women but reported less 
growth than women in critical skill areas commonly associated with the ability to 
discuss well. They ultimately focused in on two or three items that they were 
interested in working on for the following year. They did this in faculty cohort 



groups. 



In addition to the large group presentation, each instructor was given a 
printout of his or her results and how those results compared to the overall group 
means. Individual faculty members were allowed to use these results however 
they saw fit. Some contacted the researchers for follow-up data, some 
contacted the researchers to have statistical terms re-explained and some filed 
their reports but did not look at them again. While the first two responses were 
more pleasing to the researchers, the freedom to respond in a variety of ways 
allowed assessment to be seen as less heavy-handed and more accomodating 
to faculty needs, time constraints, level of knowledge about measurement and 
numbers and individual philosophies regarding the nature of teaching and 
learning. Cross and Angelo (1984) wrote that classroom assessment should 
respect the autonomy, academic freedom and professional judgment of faculty. 
Decisions on how to respond to information gained from assessment techniques 
should be made by the individual faculty member if assessment is to be 
endorsed by faculty and integrated into their role in the university. 

At the end of the presentation, the faculty asked intelligent questions and 
provided thoughtful suggestions for analyticalmethods. They wanted to continue 
the survey as well as interviews or focus groups. A Core faculty member from 
psychology, with a background in structural equation modeling using survey 
data, joined the assessment team to help with analysis. He and more than thirty 
Core faculty from various departments are now stakeholders in the assessment 
process. 
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Additional presentations of data were given to the institution's retention 
committee, the administrator's forum, the Honor's college and the chapel 
concerns' committee. "One of the most important benefits of assessment is the 
data's ability to raise the critical questions, thereby setting the institutional 
discussion and decision agenda. . . .faculty-administrative conversations, 
grounded in the data, provide the primary opportunities for university leadership 
to effect positive change" (Young and Knight, 1993, p. 29) At the end of the first 
year, the interplay between faculty and administrators, brought about by the 
inclusion of the Core assessment in the survey of first year students' satisfaction, 
was beginning to create an environment in which assessment was taken 
seriously and used more frequently in day-to-day decision making. Faculty were 
engaged and had become stakeholders in the assessment of their own program 
and in the assessment of how that experience fit into the larger experience of 
first year students. 

Conclusions 

Key elements of this project’s success in developing faculty as 
stakeholders in assessment include: 1) creating an assessment team consisting 
of staff and faculty, across disciplines, and from insider and outsider 
perspectives; 2) developing trust and furthering collaboration by listening to and 
respecting student and faculty viewpoints; 3) following the natural pace and 
rhythm of faculty (meetings after graduation, for example); 4) addressing faculty 
concerns and building consensus behind the scenes; and 5) once they become 
stakeholders, letting faculty adapt an assessment program to meet their needs. 
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